How to input HDFS file into R mapreduce for processing and get the result into HDFS file -

-

I have the same question as the link below in the stack overflow

I, the location "/somnath/logreg_data/ds1.10.csv" read one file from HDFS to reduce the number of columns from 10 to 5 and then another place "/ somnath / logreg_data / reduced / ds1. 10 am tring to write. At least CDSV is using the transfer.csvfile.hdfs.to.hdfs.reduced function below in HDFS. 

  transfer.csvfile.hdfs.to.hdfs.reduced ("HDFS: //10.5.5.82: 8020 / Somnath / logreg_data / ds1.10.csv", "HDFS: //10.5 .5.82: 8020 / Somnath / Logreg_data / low / DS1.10.rated.csv ", 5)   
 The function definition is  
  transfer.csvfile.hdfs .to.hdfs.reduced = function (hdfsFilePath, hdfsWritePath, reducedCols = 1) {# local.df = data.frame () # hdfs.get (hdfsFilePath, local.df) # to.dfs (local.df) # R .file & lt; - hdfs.file (hdfsFilePath, "r") transfer.reduced.mapped = function (., M) {label & lt; - M [, dim (M) [2]] weak. Plotter & lt; - M [, 1: reduced calls] less. M & lt; - cbind (reduced predictors, labels) keyless (1, as.Numeric (reduced.M))} less. Value = value (from.dfs (mapreduce (input = From.dfs (hdfsFilePath), input.format = "native", map = function (., M) {label & lt; - m [, dim (m) [2] ]] Less printed (labeled) .lotter <> - M [, 1: reduced calls] Less. & Lt; - cbind (weak pictures, label) keval (1, as.numeric (reduced.M)) }))) Write writeable (lower value, file = "/ root / somnath / less D.values.csv") w file and lieutenant; - hdfs.file (hdfsWritePath, "w") hdfs.write (reduced.values, w.file) # to.dfs (reduced.values)}   paste (if (read.) " R "and" w ", if (format $ mode = =: can not open connection call: transfer.csvfile.hdfs.to.hdfs.reduced ... make.keyval.reader -> Do.call - & gt; ; & Lt; Anonymous & gt; - & gt; In addition to the file: Warning message: in the file (fname, paste (if (read.) "R" else "w", if (format $ mode ==: '  
  OR   
 When I'm trying to load down A file from the HDFS using the commands I get, I get the following error:  
  & gt; x & lt; - hdfs.file (path = "hdfs: //10.5.5.82 : 8020 / Somnath /logreg_data/ds1.10.csv",mode="r ") Error in hdfs.file (path =" HDFS: //10.5.5.82: 8020 / Somnath / logreg_data / ds1.10.csv " Any effort to apply will be highly appreciated  
 thanks   
 
 < P> Actually I found the problem described above.  
  r.file & lt; - hdfs.file (hdfsFilePath, "r") from.dfs (mappedead) (input = as.matrix (hdfs.read.text.file (r.file)), input.format = "csv", map = .. The complete revised function is below:  
  transfer.csvfile.hdfs.to.hdfs.reduced = function (hdfsFilePath, hdfsWritePath, reducedCols = 1) {hdfs.init () # local.df = data.frame () # hdfs.get (hdfsFilePath, local.df) # to.dfs (local.df) r.file & lt; - hdfs.file (hdfsFilePath, "r") transfer.reduced.map = function (., M) {numRows & lt; - Length (M) M.vec .elems & LT; -unlist (lapply (m, function); strsplit (x, "")) M.matrix & lt; - Matrix (M.vec.elems, Nrow = numrows, byrow = TRUE) Labels & LT; - M. Metrics [, dim (M. matrix) [2]] weak. And lie down - M. matrix [, 1: reduced calls] less. M & LT; - CBIID (weak pictures, label) as keyval (1, .numeric (reduced.M))} less. Value = value (from.dfs (mappedreadus (input = as.matrix (hdfs.read.text.file (r.file)), input.format = "csv", map = function (., M) {numRows & lt ; - Length (M) M.V.C.LX & lt; -list (lapply (m, function x) strsplit (x, ","))) M. matrix <- Matrix (M.vec.elems, Labels & lt; - M. matrix [, dim (M. matrix) [2]] Weak. Article. & Lt; - M. matrix [, 1: reduced call] Less. M & LT; - cbind (decrease D.predictors, label) keyval (1, as.numeric (reduced.M))}))) write.table (lower value, file = "/ root / somnath / short. Values.csv ") w.file & lt; - hdfs.file (hdfsWritePath, "w") hdfs.write (lower value, w.file) hdfs.close (r.file) hdfs.close (w.file) # to.dfs (reduced.values)} < / Code>  
 Hope this helps and do not forget to give points if you find it useful. Go

Comments

Post a Comment

Popular posts from this blog

ios - Adding an SKSpriteNode to SKScene from a child SKSpriteNode -

-
I have a SKScene where I am adding an SKSpriteNode I have subclassed the SKSpriteNode class to create this node. In the subclass I am defining some SKActions on Sprite. What do I want to do when the end of this phantom ends the SKAction sequence, then I add a new Prayer node to this scene. How is this possible. The following code is mine: The code for the sequence is that I'm running on the Scansprinitode subclass (TEMissileNode): - SKAction * moveDown = [SKAction moveToY: Self.position.y - 20 Duration: 0.2]; SKAction * animation = [SKAction animateWithTextures: textures timePerFrame: time / 7]; SKAction * moveMissileProjectile = [SKAction Moving: Pointoffscreen Term: Time]; SKAction * group = [SKAction group: @ [animation, movement projectile]]; SKAction * Sequence = [SKAction Sequence: @ [Hilldown, Group, [SKAction removalFromParent]]]; [Self run action: sequence]; From the main scene, I call those actions that execute these tasks TEMissileNode * missile = [tmissil...
Read more

Matlab transpose a table vector -

-
An embarrassing simple question seems to be, but how can I move a Matlab table vector? aTableT = aTable '; I tried a standard syntax for the simple interaction of a line vector to a line vector aTable : ATableT = reshape (aTable, 1, Height (aTable)); and aTableT = rot90 (aTable); According to the head, the last time table should work for table, see. However, I get this error code: Error type in using table / permit (line 396) Undefined function 'permute' for the input arguments. Error in rot 90 (line 29) b = magnitude (b, [2 1 3: ndims (a)]); NB: fliplr is not useful either pretty sure I have covered clear angles - any ideas? Thanks! Try changing your table into an array, move it, then back to a table In other words, try doing this: aTableArray = table2array (aTable); ATableT = array 2table (aTableArray. '); I have also read the document for rot90 , and it says that rot90 is definitely working for tables Do, and I find you...
Read more

c# - Textbox not clickable but editable -

-
I have a small form with 10 text boxes, I have set them in the correct tab order. I want them on my tab service I was wondering if there is a way to set the text box so that they can be selected for editing until they are tabbed. That is ... I do not want the end user to edit the text box to click on them, I just want to make them editable through tabu. this should do the trick public partial category poor text box: Text Box {Safe Override Zero WndProc (Ref: Message Message) {if (m.msg == (Int) WM.LBUTTONDOWN) {Return; // mount down events} base.WndProc (ref m); }} Window message enum can be found. How to do this without anherizing: text box : class EatMouseDown: netwindows {safe override zero WndProc (ref message message) {if (m.msg == (int) WM.LBUTTONDOWN) {return; } Base.WandProc (Ref M); }} Safe Override Zero Onload (EventEurge E) {base.OnLoad (e); How to do this without any part: left a clean part, whatever is important, this can be a buggy but it works R...
Read more